You can find the code and data for this article at this link. It's all hosted on Deepnote, a new kind of data notebook designed for collaboration. Deepnote has become my home in the cloud for all of my data science work. Thank you to Deepnote for sponsoring this week's

The first quarter delivered home-sale profits that were more than twice as high as anticipated, the company said. Zillow expected to make money primarily from transaction fees and from services such as title insurance--not from making a killing on the flip. The company's algorithm, which was supposed to predict housing prices, didn't seem to understand the market. Zillow was also behind on its target for home purchases. By the summer, it had the opposite problem, the company later acknowledged.

In February, Zillow appeared so confident in its ability to use artificial intelligence to estimate the value of homes that it announced a new option: for certain homes, its so-called "Zestimate" would also represent an initial cash offer from the company to purchase the property. The move, touted by a company exec at the time as "an exciting advancement," was intended to streamline the process for homeowners considering selling to Zillow as part of its home-flipping business. Zillow promoted this option as a way to make it convenient to sell a home while minimizing interactions with others during the pandemic. Just eight months later, however, the company is shutting down that business, Zillow Offers, entirely. The decision, announced last week, marks a stunning defeat for Zillow.

Zillow, an online marketplace that facilitates the buying, selling, renting, financing, and remodeling of homes, employs lots of AI technologies to do things like estimate home prices. But the output of AI systems like these can be opaque, creating a "black box" problem where practitioners and customers can't audit the systems properly. Without transparency, serious problems like algorithmic bias can persist undetected, and trust in the models becomes impossible. For obvious ethical reasons, this is why explainable AI (XAI) is so crucial to the creation and deployment of AI systems, but pragmatically, it's also key to the success of AI-powered products and services from companies like Zillow. David Fagnan, director of applied science on the Zillow Offers team, discussed with VentureBeat how and why XAI is indispensable for the company.

I chose the Bay Area housing price dataset that was sourced from Bay Area Home Sales Database and Zillow. This dataset was based on the homes sold between January 2013 and December 2015. It has many characteristics of learning. The dataset can be downloaded from here. There are several features that we do not need, such as "info", "z_address", "zipcode"(We have "neighborhood" as a location variable), "zipid" and "zestimate"(This is the price estimated by Zillow, we don't want our model to be affected by this).

For my final project at Metis, I wanted to choose something that enabled me to incorporate all that I had learned during the past three months. Predicting Portland home prices allowed me to do this because I was able to incorporate various web scraping techniques, natural language processing on text, deep learning models on images, and gradient boosting into tackling the problem. Below you can see 8,300 single family home sales that I scraped in Portland, OR between July 2016 - July 2017. Obviously, neighborhood plays a large role. The West Hills (in red) are one of the priciest areas in town, whereas East Portland is cheaper.

Recently, the Urban Institute called for the creation of "neighborhood-level early warning and response systems that can help city leaders and community advocates get ahead of (neighborhood) changes." Open data and open-source analytics allows community stakeholders to mine data for actionable intelligence like never before. The objective of this research is to take a first step in exploring the feasibility of forecasting neighborhood change using longitudinal census data in 29 Legacy Cities (Figure 2). The first section provides some motivation for the analysis. Section 3 provides results and the final section concludes with a discussion of community-oriented neighborhood change forecasting systems. Neighborhoods change because people and capital are mobile and when new neighborhood demand emerges, incumbent residents rightfully worry about displacement.

Great, you've just identified some of the characteristics of a good generalization: a relatively simple, abstract, less detailed, model that is consistent with (or fits, or explains) the observations, and is more broadly applicable beyond the cases you have seen. That's how humans think, but how does this help us design a computer algorithm that generalizes well? This helps us in a couple of ways. Firstly, it gives us a framework for designing an ML algorithm. Just like humans build mental models based on their observations, an ML algorithm should ingest training data and output a model that fits, or explains the training data well. A model is the mathematical analog to the human idea of a "concept" or "mental model"; it's the mathematical formalization of what we have been informally calling "rules" until now. A model is essentially a function takes as input the characteristics (i.e.